This invention relates generally to image-based rendering (IBR), and more particularly to determining a minimum sampling rate and/or a minimum sampling curve for IBR.
Image-based rendering (IBR) in general simulates a continuous range of virtual camera viewpoints of a scene from a discrete set of input images of the scene. IBR can be used for a variety of different applications. For example, a currently popular application is to use IBR in conjunction with electronic commerce-type applications on the Internet. A user interested in seeing a car""s interior or exterior from different viewpoints, for example, may be able to rotate inside a virtual camera inside the car""s interior, and zoom in and out to see more or less of the interior, and may also be able to rotate a virtual camera around the car""s exterior. Thus, using IBR in such instances enables consumers to better see physical real-world objects that they may desire to purchase, allowing those consumers to make better-informed decisions.
IBR can be classified as a continuum between two different approaches. At one end, traditional texture mapping relies on accurate geometrical models, but only a few images. For example, in an IBR system with depth maps, such as three-dimensional (3D) warping, view interpolation, view morphing, and layered-depth images, a model has a set of images of a scene and their associated depth maps. When depth is available for every point in an image, a new image can be rendered from any nearby point of view by projecting the pixels of the image to their appropriate 3D locations, and reprojecting them onto a new image.
At the other end of the continuum, light-field rendering uses many images, but does not require any geometrical information. For example, light-field rendering generates a new image of a scene by appropriately filtering and interpolating an already acquired set of sample images. An approach known as Lumigraph is equivalent to light-field rendering, but applies approximated geometry to compensate for non-uniform sampling to improve rendering quality. Light-field rendering, however, has a tendency to rely on oversampling to counter undesirable aliasing effects in the rendered new image. Oversampling results in intensive data acquisition, significant storage space required to store the acquired images, and a large degree of redundancy in the acquired images.
Little is known in the prior art as to the relationship between IBR techniques at the texture-mapping end of the continuum and IBR techniques at the light-field rendering end of the continuum. In particular, when a given depth is specified, the number of sampled images necessary to achieve optimal new image rendering is unknown. For light-field rendering such as Lumigraph, the number of sampled images necessary to achieve optimal new image rendering is also unknown, which results in prior art approaches oversampling the scene, as described in the previous paragraph. For these and other reasons, therefore, there is a need for the present invention.
The invention relates to determining a minimum sampling rate for light-field rendering specifically, and to determining a minimum sampling curve for image-based rendering (IBR) in general. In one embodiment, the minimum sampling rate for light-field rendering is determined in accordance with             Δ      ⁢              xe2x80x83            ⁢              t        max              =          1              2        ⁢                  K                      Ω            v                          ⁢                  xe2x80x83                ⁢                  fh          d                      ,
where Kxcexa9v accounts for a light-field signal cut-off frequency, a sampling camera resolution and an output resolution, ƒ specifies a sampling camera focal length, and hd specifies a light-field depth range. In another embodiment, the minimum sampling curve for IBR is determined as constrained by Nd=2Kxcexa9vƒhdxcex94t, Ndxe2x89xa71, where Nd is the number of depth layers, xcex94t specifies a sampling interval along a t direction, Kxcexa9v accounts for a signal cut-off frequency, a sampling camera resolution and an output resolution, ƒ specifies a sampling camera focal length, and hd specifies a depth range. Where IBR is to be performed under uncertain depth conditions, the minimum sampling curve is determined in one embodiment as             Δ      ⁢              xe2x80x83            ⁢              t        max              =                  min                  z          e                    ⁢              xe2x80x83            ⁢                                    (                                          z                e                            +                              Δ                ⁢                                  xe2x80x83                                ⁢                η                                      )                    ⁢                      xe2x80x83                    ⁢                      (                                          z                e                            -                              Δ                ⁢                                  xe2x80x83                                ⁢                η                                      )                                    4          ⁢                      xe2x80x83                    ⁢                      fK                          Ω              v                                ⁢                      xe2x80x83                    ⁢          Δ          ⁢                      xe2x80x83                    ⁢          η                      ,
where xcex94tmax specifies a maximum sampling interval along a t direction, Kxcexa9v accounts for a signal cut-off frequency, a sampling camera resolution and an output resolution, ƒ specifies a sampling camera focal length, ze specifies an estimated depth, and xcex94xcex7 specifies a depth error.
Embodiments of the invention provide for advantages over the prior art. In particular, embodiments provide for the number of image samples from a four-dimension (4D) light field and the geometrical and textural information needed to generate a continuous representation of a scene for which new images are to be generated. This is also referred to as plenoptic sampling, that is, the number of images needed for plenoptic modeling. Furthermore, the minimum sampling curve described in some embodiments of the invention serves as a guiding design principle for IBR systems, and bridges the gap between IBR and geometry-based rendering.